Authentication against a Jazz application is quite complex. When using the Plain Java Client Libraries, this is not a concern, as the ITeamRepository handles all that for you. It authenticates against the server and keeps the authentication valid over time. This is important, because application servers have timeouts that can void the the authentication of a session. A very common timeout that impacts the Jazz applications is the Liberty Profile default of 2 hours.
Running a client application against such a server requires detecting the need to authenticate and performing the authentication as well as to guarantee the current request is performed. Enterprise applications also require to work in complex setups supporting different ways to authenticate. Some examples are authentication based on Liberty Profile file based user management, LDAP based delegated authentication and user management, Jazz Authentication Server (JAS) based authentication, Kerberos and more. Implementing a client that uses the REST and OSLC APIs directly requires to develop a mechanism that handles this automatically.
Warning, this is complex!
I have tried to simplify this topic as good as possible, within my limited knowledge and experience, however this blog turned out to be very lengthy and complex. I also had to redo my screenshots multiple times to correct for changes during writing the post. Again, this ended up not being the casual blog post. You have been warned.
Approaching authentication
I have worked with the EWM Plain Java Client Libraries in the past. They provide a standard login mechanism to use and that keeps you logged in. I worked with Eclipse Lyo and used the mechanism provided with it, similar to plain Java API. I started just using an HTTP Client and I implemented a simple Login mechanism that was able to perform a Form based authentication. There was no code that would keep me logged in. When I started experimenting with Python, I just implemented a similar login mechanism that uses Form based authentication as explained in this post. In the first steps of the script, the authentication was executed, under the assumption that it was valid for the rest of the script.
That was sufficient for my experiments with my local test server. However it does not work with systems that have JAS enabled. This created questions, such as which authentication method was available and how to detect that, that required deeper knowledge about how the authentication was supposed to work. This approach also does not work for scripts that have to run longer durations. In many environments, the authentication expires after some timeout.
I started reading the Native Client Authentication article from the Team Wiki to understand how it was supposed to work. I got confused. I talked to some of my peers and hoped for someone having looked into this already. No luck. I cite the answer of a very skilled colleague that I hold in high regard: “Ah, this Native Client Authentication article is the most cryptic article I have ever come across.. at times it appears like everything has been told and at almost the same time, it appears like it is telling us nothing” and “I read the article now the 1001th time, and I still don’t understand”. Exactly.
I could not leave this as the answer and started coding while trying to follow the article. It makes no sense to me, to try to rewrite the article in this blog. Also just words will not bring any clarity. It looks as if language is not clear enough to describe what needs to be done.
The only way I am able to explain what goes on is showing code samples. I did not want these blogs to be about code, but it seems to be unavoidable to get clarity and to explain what goes on using the resulting code in my library class ELMCommLib. At this time I will use images to show the code. I will likely make the code available as open source later. Once done, I will link the download location to the blogs, as usual. The code is the result of a quite lengthy development and test process. It only implements parts of what the Team Wiki article explains. I have only so many test systems available and I can only test what these systems do. So here the disclaimer…
Disclaimer
Please note that so far I have only tested against a standard system supporting Form based authentication and a system supporting JAS. I used a trick to test the fallback to Basic authentication. My general disclaimer to the right of the page applies too. I’m not responsible for any damage done when following this article.
Context of the blog post is the series
This is the series of planned posts I intent to publish over time. Most of the examples will be EWM based, but quite a lot of the content applies to more ELM applications. The examples where performed with versions 6.0.6.1 and 7.0.x.
- Using the EWM REST and OSLC APIs
- ELM Authentication – this post
- EWM Discovery
- EWM Work Item OSLC CM API
- EWM OSLC Query API
- EWM REST API to access existing Work Item Queries
- EWM Reportable REST API
Where to authenticate
To authenticate to ETM and EWM, use the context root of the server e.g. use https://elm.example.com:9443/qm or https://elm.example.com:9443/ccm as the destination for the authentication.
For DNG/ERM and JTS use the context root of the JTS e.g. https://elm.example.com:9443/jts to authenticate.
OAuth and Jazz Authentication Server
There is another type of authentication called OAuth. Please see ELM Applications Authentication and Authorization via OAuth 2.0 Workflow and related topics for this.
Also see OAuth2.0 Access flow on ELM Configured with a SAML IDP or OIDC Provider for other options.
See Jazz Authorization Server – Landing page for more details.
How the authentication code works
I started with creating a function internalJazzRequestWithAuthentication() that can be used for all HTTP requests by passing the method and all parameters needed. This function deals with the whole authentication requirements and makes sure the authentication dance is performed when needed. I provide more convenient functions that wrap this function and are easier to use.
This function has become long and complicated, so I have to show it in smaller pieces below.
The function delegates the request to the function internalJazzRequest() which is performing the correct call using the requests library. The function looks like this.
The function returns the result for the request. The result is now evaluated in the first part of the authentication code of internalJazzRequestWithAuthentication().
The function internalJazzRequestWithAuthentication() checks the status code of the response to be able to determine if anything related to authentication needs to be done.
The first HTTP status code it evaluates is 200. Usually 200 means the call went OK and the result is the desired result. However, the result headers can contain information that the access to the resource would require authentication and the call actually did not return what was expected.
The function internalHandleFormChallenge() is used to handle this case. It checks the result headers to figure out if Form based authentication is required. The code to detect the Form authentication request is easy enough to understand. The code checks for the presence of a special custom header X-com-ibm-team-repository-web-auth-msg. If the header is present and contains the value authrequired a Form based authentication is required.
If Form based authentication is needed, the code performs the login and retries the initial call. Note that currently the code uses the URI that is provided when creating the communication library for authentication. This needs to be changed at some point. The code has to get the public URI root from the result. This is important for applications like DNG that delegate authentication to JTS.
The function also handles the case that the initial request was done for an insecure URI. This is handled as a fail and a false is returned.
If no authentication was required, the current result is returned back.
If form based authentication is required, the function internalLoginForm() is used to perform the form based authentication. It is essentially the legacy code remaining from an initial login mechanism that I used for automation examples. There is a façade function loginForm in the code that keeps this code compatible to old scripts. The form based authentication sends the credentials to the URI publicURI/j_security_check and checks if the authentication worked by looking at the response location header.
Back in the function internalHandleFormChallenge() the code tests if the login was successful. If so, the initial request for the resource is re-executed in the function internalRetry() and the result is returned. The retry code will be explained later.
There are several other return codes, typically for different methods like POST, PUT etc. that can require form based authentication in internalJazzRequestWithAuthentication(). One example is return status code 302 which is handled exactly like the code 200.
The wiki page Native Client Authentication mentions the return status codes 303 and 307. None of the test systems I worked against ever responded in any of those codes. At the time of writing they are detected in a check and the call will fail for these codes. Once I start sending unauthenticated PUT, POST, DELETE or PATCH requests, I might see these response codes. The handling would be the same. One of the next steps for me will be to setup tests that force the other situations.
The last possible response is 401. This is the most complex case. It handles all other cases for authentication, especially Jazz Authorization Server (JAS), OICD and some other authentication options. See the initial part of the handling for 401 in internalJazzRequestWithAuthentication() below.
As explained in the wiki page it is necessary to check for the header WWW-Authenticate. If this header is present, it needs to be evaluated. This could be a Basic authentication challenge or a Bearer challenge for Jazz Authentication Server/OIDC/OAuth. The situation where the Bearer or the Basic challenge is missing is handled at the very end.
If the Basic challenge and the Bearer challenge is detected, the headers X-JSA-AUTHORIZATION-REDIRECT and X-JSA-AUTHORIZATION-URL are evaluated to determine the authentication strategy. For the test system I have available, I needed to follow the JAS authentication.
The first step to do JAS authenticate with Basic and Bearer challenge is to get the redirection URL from the header X-JSA-AUTHORIZATION-REDIRECT. Based on this the code constructs the first redirection URL stored in xjsaauthredirect with the additional parameter '&prompt=none'.
The code performs a GET on the no-prompt redirect URL. The result of this call determines the next steps. When the GET does not return 200, the authentication is still valid and the code runs a retry for the original request, the result of which is returned. This is shown here.
If the GET returns status code 200 authentication is required. The code below shows the code to handle the authentication. Note, that there is some code that gets information that remain unused at this time.
To reauthenticate the code performs a GET on the redirection URI in xjsaauthredirect, this time without the additional no-prompt parameter. The request can result in a status code 401. This is indicating Kerberos or some other authentication method. I have not implemented this code path at the moment. I lack a test system that behaves this way.
The request can result in a status code 200. In this case we are on the OAuth path. The code performs a request on the redirect URI in xjsaauthredirect, this time providing an additional authentication header conforming to the Basic authentication header. The header is computed in the call to HTTPBasicAuth(). If this succeeds with a status code 200, the client is authenticated and the code retries the initial request. This is not strictly necessary for all requests to do that, but for now, I wanted to do it anyway. The result is returned back to the outer code. Note that the Authentication header is only sent this one time. In contrast when using Basic authentication like below, the Authentication header is sent with all the subsequent calls.
In the case that the server only returned the Basic authentication header, the code below initializes the normal Basic authentication mechanism. The code is written in an way that now always sends the Basic authentication header with all requests in this case.
The case that Kerberos was detected is also handled here at the end. Lacking a test system the code returns unsupported and False for now. When the code was unable to identify the authentication method, e.g. the server was not a Jazz server, the code fails as well.
This is what I was able to come up for handling authentication. There is no guarantee, that it is completely correct. I am lacking test systems that behave in different ways, so testing is very limited.
Please note, that the authentication is only valid for the session that is used and based on several cookies that are managed by the session. Also note, as already mentioned, the session can be invalidated by the server at any time and the authentication would have to (automatically) be done again.
Retry code
I ended up creating a function to handle the retry for the initial call that led to the authentication challenges. At the moment the code for GET is tested and works. The code currently basically reruns the initial request and checks if that was successful. Then it returns True and the result or False.
While writing this code, it appears that the retry code might have to be changed later. When the first call requiring authentication is something other than GET the code might fail. As an example, I am not sure at the moment how a POST e.g. for creating a work item would be handled in detail. The key here is that POST of a work item may only be run once. This is going to be an interesting journey and there is much to be learned about redirects and retries. Once all cases are tested, I will try to re-upload the changes.
Example for authentication flows – Form authentication
As already mentioned, I am able to track the communication in my framework. The following sequence of messages is related to a GET request against a system that uses Form authentication. Please ignore the files with extension .nt, they are not relevant for the authentication.
The first call is to get the rootservices document. The rootservices document is not protected, so no authentication happens.
The next call is to get the service provider catalog which is discovered in the rootservices document. The service provider or workitems catalog is a protected resource, this means the request requires authentication. The request below shows that the call was redirected from the work items catalog to the web page showing the login prompt. The page code in my logs usually shows some mysterious message that “Javascript is either disabled or not available in your Browser”. This is based on the fact that there is no browser that sent the request and all JavaScript code in the HTML page fails.
The response header also contains the header X-com-ibm-team-repository-web-auth-msg authrequired indicating authentication is required.
The code detects that authentication is required. And executes the form based authentication, providing the username and password unencrypted. The latter is the reason why it is absolutely necessary to use HTTPS and not HTTP as protocol. HTTP sends the credential information unencrypted. The response of the login call contains the redirection location to the previous call.
At the moment, the code shown here does not use the redirect, it uses the original resource URI to retry the call. This is the retry the code performs. Please note that the destination URL and the Request URL are now the same. The response body contains the protected resource service provider catalog. This catalog contains the information about the visible available project areas.
All subsequent calls in this session are performed without any authentication challenge. The next call to get the work item service for the desired project area is omitted.
Example for authentication flows – JAS authentication
Finally lets have similar look at the sequence of events when running against a system with JAS Authentication. Note, I have replaced the original host name with a generic one, should there be any unclear references, in case I did not spot them. The following flow is the result of the same version of software running against a different URL that has JAS authentication configured.
The first call is to get the rootservices document again. The rootservices document is not protected, so no authentication happens.
The next call is to get the service provider catalog which is discovered in the rootservices document. The service provider/work item catalog is a protected resource, this means the request requires authentication. The request below shows that the call to the service provider catalog fails with a status 401. The response header also contains the header WWW-Authenticate with the form containing the Basic and Bearer challenge. The headers X-JSA-AUTHORIZATION-URL and X-JSA-AUTHORIZATION-REDIRECT are also present. This provides with the authentication and redirect URL for the authentication.
The next step is the GET request against the redirect URL with the additional parameter &prompt=none. This request collects additional information after some redirects. I don’t claim I understand the details here. Key is that the call can either determine that there is actually a valid authentication already, or a new authentication is required.
Here the previous call results in status code 200. This means that JAS authentication is required. So we GET the previous target URL, but without an extra parameter &prompt=none.
Note that the request gets redirected to the login URL that we have seen in the response to the initial call in the header. X-JSA-AUTHORIZATION-URL. The response header seems to indicate that Form based authentication is a valid fallback: X-com-ibm-team-integration-jazzop-auth-msg form-login. I am not sure and have not tried. I followed the wiki page Native Client Authentication implemented the JAS authentication. The response body here is the HTML code of the login page. If the client was a browser, it would be possible to show it to the user.
The previous request ended with a status 200 which means JAS/OAuth2 is the way to go. This is done in the next step. The request is executed again and an additional Basic authentication header is created and sent with the request. Note that the Authentication header is not shown below. If it was I would have had to redact it, because the data is only encoded, not encrypted. This is also the reason why Jazz Applications are by default all using HTTPs only. Do not send your credentials over HTTP.
Also note that the request gets successfully redirected back to the service catalog URI. The response body actually contains the service provider/work items catalog.
Because it is just a GET request, It would be possible to just return it as a result. At the moment I run a last request to retry the initial get. I will leave that request out in this example.
Summary
As always, I hope that this blog helps users out there that work with the Jazz products and their APIs. This was a very complex topic and I hope I was able to shed some light based on the experience I have gained over time. There are several cases missing, which I was unable to test lacking access to test systems and also test code that I have not yet developed. I will try to keep the post updated as always. Once I have finished this series, I will also try to publish the code for re-use.
If you have made your own experience with this topic and want to share, please do so in the comments.
rsjazz 
I did something similar in Python, but only got as far as form-based. It’s great to see a fuller implementation here that I can reference. Thanks! Yes, the Native Client post is a beast.
I will look into making the code available some time in the future. No definitive plans though.
I found this https://github.com/IBM/ELM-Python-Client – so I think I’m off and running. 🙂
Good find!
I am stuck on this. Details here: https://jazz.net/forum/questions/284065/oidc-connection-to-ibm-elm-saas-environment. Any guidance is greatly appreciated.
I responded on your question.
Nice Blog, thank you!